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Abstract 

This report includes an assessment of the network performance in terms of lost observing time 
for the 2012 calendar year. Overall, the observing time loss was about 12.3%, which is in-line with 
previous years. A table of relative incidence of problems with various subsystems is presented. The 
most significant identified causes of loss were electronics rack problems (accounting for about 21.8% 
of losses), antenna reliability (18.1%), RFI (11.8%), and receiver problems (11.7%). About 14.2% of 
the losses occurred for unknown reasons. New antennas are under development in the USA, Germany, 
and Spain. There are plans for new telescopes in Norway and Sweden. Other activities of the Network 
Coordinator are summarized. 


1. Network Performance 

The overall network performance was for the most part good. This year we return to reporting 
a detailed assessment, which was not provided for the two previous years. 

This network performance report is based on correlator reports for experiments in calendar year 
2012. This report includes results for the 148 24-hour experiments that had detailed correlator 
reports available as of March 8, 2013. The data set examined includes approximately 500,000 
dual frequency observations. Results for 16 experiments were omitted because either they were 
correlated at the VLBA, they have not been correlated yet, or correlation reports were not available 
on the I VS data centers. Experiments processed at the VLBA correlator were omitted because 
the information provided for them is not as detailed as that from Mark IV correlators. The 
experiments that have not been correlated or do not have correlator reports available yet include 
some JADE, JAXA, OHIG, R&D, T2, and EUR experiments. In summary, roughly 90% of the 
data from scheduled 24 hour experiments for 2012 are included in this report. That is similar to, 
and actually a little more complete than, the coverage of reports for previous years. 

An important point to understand is that in this report, the network performance is expressed 
in terms of lost observing time. This is straightforward in cases where the loss occurred because 
operations were interrupted or missed. However, in other cases, it is more complicated to calculate. 
To handle this, a non-observing time loss is typically converted into an equivalent lost observing 
time by expressing it as an approximate equivalent number of recorded bits lost. As an example, 
a warm receiver will greatly reduce the sensitivity of a telescope. The resulting performance will 
be in some sense equivalent to the station having a cold receiver but observing for (typically) 
only one-third of the nominal time and therefore recording the equivalent of only one-third of 
the expected bits. In a similar fashion, poor pointing can be converted into an equivalent lost 
sensitivity and then equivalent fraction of lost bits. Poor recordings are simply expressed as the 
fraction of total recorded bits lost. 

Using correlator reports, an attempt was made to determine how much observing time was lost 
at each station and why. This was not always straightforward to do. Sometimes the correlator 
notes do not indicate that a station had a particular problem, while the quality code summary 
indicates a significant loss. Reconstructing which station or stations had problems — and why — in 
these circumstances does not always yield accurate results. Another problem was that it is hard 
to determine how much RFI affected the data, unless one or more channels were removed and that 
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eliminated the problem. It can also be difficult to distinguish between BBC and RFI problems. 
For individual station days, the results should probably not be assumed to be accurate at better 
than the 5% level. 

The results here should not be viewed as an absolute evaluation of the quality of each station’s 
performance. As mentioned above, the results themselves are only approximate. In addition, some 
problems such as weather and power failures are beyond the control of the station. Instead the 
results should be viewed in aggregate as an overall evaluation of what percentage of the observing 
time the network is collecting data successfully. Development of the overall result is organized 
around individual station performance, but the results for individual stations do not necessarily 
reflect the quality of operations at that station. 

Since stations typically observe with more than one other station at a time, the average lost 
observing time per station is not equal to the overall average loss of VLBI data. Under some 
simplifying assumptions, the average loss of VLBI data is roughly twice the average loss of sta- 
tion observing time. This approximation is described in the Network Coordinator’s section of 
the I VS 2001 Annual Report. For 2012, this agrees reasonably well with the actual number of 
(single frequency: S or X) single baseline observations on which the correlator reported failure, 
approximately 21.1%, but other factors, particularly the dual frequency nature of useful geodetic 
observations, complicate the picture. For 2012, the actual percentage of data (dual frequency) that 
was not included by the analysts was approximately 28.1%. This is even larger (by approximately 
34%) than the single baseline observations reported lost by the correlator. It is expected that this 
number should be higher because the analysts use additional criteria beyond what is discussed 
here to decide when to exclude observations. However, it means in effect that only about 72% of 
the observations we attempted to collect were useful. 

For the 148 experiments from 2012 examined here, there were 1,261 station days or approxi- 
mately 8.5 stations per experiment on average. This compares to 135 experiments considered in 
the report for 2009 (the most recent year with a detailed report), which included 1,051 station days 
with 7.9 stations per experiment. The increase in the number of analyzed experiments essentially 
just reflects that the results for more experiments were available for consideration at the time the 
report was written. However the increase in the number of stations per experiment is probably 
due to a concerted effort by the IVS Coordinating Center to make the networks in the experiments 
larger. The onset of operations by the AuScope and Warkworth stations helped to make this 
possible. Of the station days for 2012, approximately 12.3% (or approximately 155 days) of the 
observing time was lost. For comparison to reports from earlier years, please see Table 1. 

The lost observing time for 2012 is more in-line with results from years before 2009. The results 
for 2009 may be artificially high due to a change in the way the results were tabulated for that 
year. We believe this year’s calculations are more in-line with how they were made before 2009. 

An assessment of each station’s performance is not provided in this report. While individual 
station information was presented in some previous years, this practice seemed to be counter- 
productive. Although many caveats were provided to discourage people from assigning too much 
significance to the results, there was feedback that suggested that the results were being over- 
interpreted. Additionally, some stations reported that their funding could be placed in jeopardy 
if their performance appeared bad, even if it was for reasons beyond their control. Last and least, 
there seemed to be some interest in attempting to “game” the analysis methods to improve station 
results. Consequently, only summary results are presented here. Detailed results are presented to 
the IVS Directing Board. Each station can receive the results for their station by contacting the 
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Table 1. Lost observing time. 


Year 

Percentage 

1999-2000* 

11.8 

2001 

11.6 

2002 

12.2 

2003 

14.4 

2004 

12.5 

2005 

14.4 

2006 

13.6 

2007 

11.4 

2008 

15.1 

2009 

21.5 

2012 

12.3 


* The percentage applies to a subset 
of the 1999-2000 experiments. 


Percentages for 2010 and 2011 are 
omitted, but should be 10-20%. 


Network Coordinator (Ed.Himwich@nasa.gov). 

For the purposes of this report, the stations were divided into two categories: large N: those 
that were included in 20 or more network experiments among those analyzed here and small N: 
those in 11 or fewer (no stations were in the 12-19 experiment range). The distinction between 
these two groups was made on the assumption that the results would be more meaningful for the 
stations with more experiments. The average observing time loss from the large N group was 
much smaller than the average from the small N group, 10.6% versus 25.7%. There are many more 
station days in the large N group than the small N group, 1,125 versus 136, so the large N group 
is dominant in determining the overall performance. 

There are 19 stations in the large N group. Nine stations observed in 50 or more experiments. 
Of the 19, eight stations successfully collected data for approximately 90% or more of their expected 
observing time. Nine more stations collected 80% or more of the time. The two remaining stations 
collected data for more than about 60% of their observing time. These results are not significantly 
different from previous years. 

There are 24 stations in the small N group. The range of lost observing time for stations in 
this category was 0%-100%. The median loss rate was approximately 11.3%, better than previous 
years. 

The losses were also analyzed by sub-system for each station. Individual stations can contact 
the Network Coordinator (Ed.Himwich@nasa.gov) for the sub-system breakdown (and overall loss) 
for their station. A summary of the losses by sub-system (category) for the entire network is 
presented in Table 2. This table includes results since 2003 sorted by decreasing loss in 2012. 

The categories in Table 2 are rather broad and require some explanation, which is given below. 

Antenna This category includes all antenna problems, including mis-pointing, antenna control 
computer failures, non-operation due to wind, and mechanical breakdowns of the antenna. 
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Table 2. Percentage of observing time lost by sub-system. 


Sub-System 

2012 

2009 

2008 

2007 

2006 

2005 

2004 

2003 

Rack 

21.8 

6.6 

8.7 

11.4 

16.3 

5.1 

6.8 

5.0 

Antenna 

18.1 

29.4 

19.2 

34.6 

19.0 

24.4 

32.9 

17.8 

Unknown 

14.2 

14.2 

17.7 

14.9 

4.0 

3.3 

10.1 

12.6 

RFI 

11.8 

5.9 

14.8 

10.4 

11.6 

6.2 

5.0 

9.3 

Receiver 

11.7 

18.6 

13.8 

14.9 

20.8 

24.2 

18.0 

25.2 

Miscellaneous 

6.9 

15.3 

12.8 

7.6 

18.0 

8.0 

8.0 

6.0 

Recorder 

5.7 

2.9 

4.1 

4.6 

3.3 

8.9 

11.1 

10.9 

Shipping 

3.6 

4.0 

5.4 

1.0 

0.0 

0.2 

1.4 

6.1 

Power 

2.1 








Operations 

2.0 

1.2 

2.3 

0.0 

2.0 

4.7 

6.1 

3.6 

Clock 

1.8 

1.9 

0.5 

0.3 

4.9 

14.5 

0.5 

3.4 

Software 

0.3 

0.1 

0.1 

0.4 

0.1 

0.5 

0.1 

0.1 


Percentages for 2010 and 2011 were not calculated. 


Clock This category includes situations where correlation was impossible because the clock offset 
either was not provided or was wrong, leading to the “no fringes” case. Maser problems and 
coherence problems that could be attributed to the Maser were also included in this category. 
Phase instabilities reported for Kokee were included in this category. 

Miscellaneous This category includes several small problems that do not fit into other categories, 
mostly problems beyond the control of the stations, such as power (only prior to 2012), 
(non-wind) weather, cables, scheduling conflicts at the stations, and errors in the observing 
schedule provided by the Operation Centers. For 2006 and 2007, this category also includes 
errors due to tape operations at the stations that were forced to use tape because either they 
did not have a disk recording system or they did not have enough media. All tape operations 
have since ceased. This category is dominated by weather and scheduling conflict issues. 

Operations This category includes all operational errors, such as DRUDG-ing the wrong sched- 
ule, starting late because of shift problems, operator (as opposed to equipment) problems 
changing recording media, and other problems. 

Power This category includes data lost due to power failures at the sites. Prior to 2012, losses 
due to power failures were included in the Miscellaneous category. 

Rack This category includes all failures that could be attributed to the rack (DAS) including the 
formatter and BBCs. There is some difficulty in distinguishing BBC and RFI problems in 
the correlator reports, so some losses are probably mis-assigned between the Rack category 
and the RFI category. 

Receiver This category includes all problems related to the receiver, including outright failure, 
loss of sensitivity because the cryogenics failed, design problems that impact the sensitivity, 
LO failure, and loss of coherence that was due to LO problems. In addition, for lack of a more 
clearly accurate choice, loss of sensitivity due to upper X band Tsys and roll-off problems 
were assigned to this category. 
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Recorder This category includes problems associated with data recording systems. Starting with 
2006, no problems associated with tape operations are included in this category. 

RFI This category includes all losses directly attributable to interference, including all cases of 
amplitude variations in individual channels, particularly at S-band. There is some difficulty 
in distinguishing BBC and RFI problems in the correlator reports, so some losses are probably 
mis-assigned between the Rack category and the RFI category. 

Shipping This category includes all observing time lost because the media were lost in shipping 
or held up in customs or because problems with electronic transfer prevented the data from 
being correlated with the rest of the experiment’s data. 

Software This category includes all instances of software problems causing observing time to be 
lost. This includes crashes of the Field System, crashes of the local station software, and 
errors in files generated by DRUDG. 

Unknown This category is a special category for cases where the correlator did not state the 
cause of the loss and it was not possible to determine the cause with a reasonable amount of 
effort. 

Interesting results for 2012 include the fact that the largest source of losses was due to “Rack” 
problems, (21.8%). About half the loss in this category is associated with DBBCs, which were in 
use at several stations in the Southern Hemisphere. The remaining rack problems were largely due 
to problems at stations with aging racks for which replacement parts are hard to find. However, 
significant improvements were made at two of the three stations with these problems by the end 
of the year. 

The next largest area of loss was “Antenna” problems (18.1%). This is down significantly 
from previous years and reflects the fact that there were no lengthy outages due to catastrophic 
antenna failures. Stations with significant antenna problems were Matera, Svetloe, Tsukuba, and 
Yarragadee. 

The “RFI” losses (11.8%) were more back in-line with years before 2009. The higher value in 
2009 may be related to differences in how the losses were treated that year. The stations with the 
most serious RFI problems were Fortaleza and Matera. 

“Receiver” sub-system problems (11.7%) was lower than in previous years. For 2012 this is 
probably due to fewer cryogenic problems, due partly to several new stations having uncooled 
receivers. 

The “Miscellaneous” category loss was smaller than previous years. This was to a small extent 
due to the fact that “Power” was broken out as a separate category this year, but this does not 
account for the majority of the change. 

Overall, while the network operated well for the most part, there are a few notable issues (in 
alphabetical order of station), while some situations improved from the previous year: 

• Fortleza RFI for channel SR4U caused almost 15% data loss for that station over the year. 
An attempt to fix this by bandpass filtering the RFI signal in the station’s S-band IF did 
not work. 

• Hobart26 is now observing without a phase calibration antenna unit. 

• Hobart, Katherine, and Yarragadee have timing issues with the DBBC back-ends. These 
cause occasional clock breaks and data gaps when they occur. The manufacturer is investi- 
gating this issue. 
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• Kokee Park’s damaged gearbox was repaired and was re-installed. This improved the an- 
tenna’s pointing and its SEFDs but did not return them to their normal levels. There are still 
problems with both azimuth gearboxes, which will need to be repaired. The station replaced 
the AC wiring going to the telescope to prevent the cryogenic compressor from tripping off. 

• The receiver at Medicina warmed up in November 2011. It is not clear when it will be 
repaired. 

• Matera’s Mark 5 samplers for S-band channels 5 and 6 have failed. Efforts are being made 
to locate replacements. Matera had an antenna failure that was repaired. 

• After completion of its bearing repair, the Noto antenna started observing again in May 
2012. Noto also repaired its BBCs, so that it has at most one or two bad BBCs now. In any 
event, it is expected that Noto will replace its aging VLBA4 rack with a DBBC in 2013. 

• Ny-Alesund’s receiver communications were repaired using the system that the TIGO station 
developed for their receiver, which had the same original design. 

• Svetloe had intermittent antenna problems that caused occasional data losses. Badary and 
Zelenchukskaya’s antenna reliability has improved. 

• TIGO has shown higher than normal SEFDs for several years. There has been no success in 
resolving this issue. 

• The Tskuba32 telescope had a major structural failure and was repaired. 

• Warkworth lost most of the scans scheduled for 2012 because of a maser failure, now repaired. 

• The Westford azimuth antenna drives continue to trip off sometimes when the site is unat- 
tended. 

• Yarragadee solved its antenna problems by installing a bandpass filter in a diplexer to elim- 
inate the need to stow the antenna when a satellite up-link is active. 

2. New Stations 

There are prospects for new stations on several fronts. These include (in approximate order of 
how soon they will start regular observations): 

• At GSFC in the USA, a new 12-m antenna has been erected and is undergoing testing. While 
this antenna is primarily for use in the development of the VGOS systems, it is expected 
that it will eventually join the network for regular observing. 

• At Wettzell in Germany, construction of the new Twin Telescope Wettzell (TTW) for VGOS 
is underway and is expected to be commissioned in April 2013. 

• At Arecibo in Puerto Rico a new 12-m antenna has been erected and is expected to be used 
for geodetic observing. 

• In Spain/Portugal, the RAEGE (Atlantic Network of Geodynamical and Space Stations) 
project aims to establish a network of four fundamental geodetic stations including radio 
telescopes that will fulfill the VGOS specifications: Yebes (1), Canary Islands (1), and Azores 
(2). 
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• In Norway, the Norwegian Mapping Authority (NMA) has received initial funding for a 
project to establish a fundamental station at Ny-Alesund, which will include a twin telescope 
of the Wettzell type. 

• Onsala has applied for funds for a twin telescope system. 

• In Russia, an effort is underway to get 12-m VGOS antennas at some of the QUASAR 
network sites. 

• Korea is planning to build one antenna primarily for geodesy (Korea VLBI system for 
Geodesy, KVG) at Sejong. There is also interest in geodetic use of the Korean VLBI Network 
(KVN), which will consist of three stations intended primarily for astronomy. 

• There is interest in India in building a network of four telescopes that would be useful for 
geodesy. 

• Saudi Arabia is investigating having a combined geodetic observatory, which would presum- 
ably include a VLBI antenna. 

• Colombia is investigating having a combined geodetic observatory, which would presumably 
include a VLBI antenna. 

Many of these antennas may become available for use in the next few years. Efforts are being 
made to ensure that these antennas will be compatible with VGOS. 

3. Network Coordination Activities 

Network coordination involved dealing with various network and data issues. These included: 

• Reviewing all experiment “ops” messages, correlator reports, and analysis reports for prob- 
lems and working with stations to resolve them 

• Responding to requests from stations for assistance 

• Providing AuScope staff with a technical operations workshop (“Mini-TOW”) for training 

• Making station visits to Hobart and Warkworth for software updates and training 

• Identifying network station issues and working with the IVS Coordinating Center and the 
stations to resolve them. This year these included: 

— Dealing with Mark 5B/5B+ “,E” scan_check errors 
— Dealing with Mark 5B/5B+ time issues 

— Preparing Mark 5 modules for use and correcting VSN problems 
— Helping stations avoid the Linux “day 49” kernel problem 
— Maintaining the FS PC kernel 

• Reviewing RFI sources, selecting bandpass filters, and providing them to the sites 

• Participating in development of the new VEX2 schedule file standard 
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• Updating RDV experiment VEX files to allow proper operation with the VLBA correlator, 
updating the notes file to reflect equipment set-up at different stations, and encouraging 
timely shipping of data 

• Recognizing and reporting DBBC issues to station observing staff 

• Reviewing Mark 5 recording error checks for problems and informing correlator staff and 
station staff 

• Assisting in troubleshooting the Kokee X-band dewar failure 

• Troubleshooting power supplies and identifying the correct parts for shipping 

• Troubleshooting video converters and organizing shipments to stations 

• Coordinating shipment of phase calibration sync trigger units to each station with a Mark 
5B/5B+ 

• Troubleshooting phase calibration issues and coordinating parts shipments 

• Providing telescope pointing analysis and advice. 

4. Future Activities 

Network coordination activities are expected to continue next year. The activities will largely 
be a continuation of the previous year’s activities: 

• Reviewing all experiment “ops” messages, correlator reports, and analysis reports for prob- 
lems and working with stations to resolve them 

• Responding to requests from stations for assistance 

• Identifying network station issues and working with the IVS Coordinating Center and the 
stations to resolve them 

• Updating Network Station configuration files 

• Planning for and teaching at TOW 2013 

• Other activities as needed. 
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